Papers with GPT-2 language model
Uncertainty and Surprisal Jointly Deliver the Punchline: Exploiting Incongruity-Based Features for Humor Recognition (2021.acl-short)
Copied to clipboard
| Challenge: | Existing work on humor recognition does not examine the actual joke mechanism . a recent study focused on humor-specific stylistic features, but few have tried to establish a connection between them and humor theories. |
| Approach: | They propose to model the set-up and punchline as part developing semantic uncertainty and disrupt audience expectations. |
| Outcome: | The proposed features can tell jokes from non-jokes, compared with baselines. |
Towards Understanding of Medical Randomized Controlled Trials by Conclusion Generation (D19-62)
Copied to clipboard
| Challenge: | Using machine learning to interpret large amounts of data can be over-whelming for clinicians. |
| Approach: | They propose to use PubMed 200k RCT sentence classification dataset to generate RCT conclusion generation task. |
| Outcome: | The proposed model improves quality and correctness in generated conclusions compared to baseline model . the proposed model is not suitable for all RCTs, but it could be improved . |
Universal Adversarial Triggers for Attacking and Analyzing NLP (D19-1)
Copied to clipboard
| Challenge: | Using adversarial triggers, a model can produce a specific prediction . adversarial attacks are useful for evaluation and interpretation . |
| Approach: | They propose a gradient-guided search over tokens that finds short adversarial triggers that successfully trigger the target prediction. |
| Outcome: | The proposed algorithm finds short trigger sequences that successfully trigger the target prediction. |
Best of Both Worlds: Making High Accuracy Non-incremental Transformer-based Disfluency Detection Incremental (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, Transformer-based text classifiers are not suitable for live incremental processing, operating only on the level of complete sentence inputs. |
| Approach: | They propose to introduce a method for word-by-word left-to-right incremental processing to Transformers such as BERT, models without an intrinsic sense of linear order. |
| Outcome: | The proposed method maintains high non-incremental performance while operating strictly incrementally. |
Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisal (2022.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based large language models are trained to make predictions about the next word by aggregating representations of previous tokens through their self-attention mechanism. |
| Approach: | They propose an entropy-based predictor that quantifies the diffuseness of self-attention and a distance-based one that captures the incremental change in attention patterns across timesteps. |
| Outcome: | The proposed models perform better over a rigorous baseline including GPT-2 surprisal than previous models that used entropy-based predictors and distance-based ones. |